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Ministry of Public and Business Service Delivery 


(GSiC) . 


Government Services Integration Cluster 
Digital Enterprise Engagement Practice (DEEP) 


CIVR Report Story Meeting 


Conversational Interactive Voice Response (CIVR) 
MVP for MTO Driver and Vehicle Information 


Monitoring and Reporting 


Digital Enterprise Engagement Practice: “Connecting the people and businesses of Ontario with their Government” 
Government Services Integration Cluster: “Enable the best experiences for citizens, business and employees” 
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MVP Reviews and Reporting (GSiC) 


Getting started our focus is the system is working and not negatively 
impacting the public or agents. Light reporting to follow. 


Calendar Who to report to 
» Feb 2 - Triage reporting for systems — full team -+ Working team 
- Feb 3 - Triage reporting for systems — full team + Owner/Support — Beth 
» Feb 6 — Reporting analysis story — content working team + Directors — Olga, Lisa 
- we will have enough calls - over 1600 + ADM’s/Associate and CIO/CCIO 
+ we can do our analysis on this together + Communications will share with DMO 
» Feb 8 - Reporting analysis story - content working team -< SWAT 


» Feb 10 - Reporting analysis story - content working team 
+ Feb 13t forward 

+ CMI calendar content meeting — include Kim and Kyle 

- Weekly full team organized by DEEP 

- Document the issues/defects in the list 


+ Measurement — Sharon Stuckless, Rene 
Alverez plus teams 
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Roles (GSiC) 


During transition we will meet regularly and these are our roles 


ServiceOntario DEEP 
e Reviewing reports — Involve BI e Initially reviewing reports 
* Reporting to leadership ° Updating the technology 
* VOC - Looking the customer journey for fixes * Update to content 

* Start with high value insights e Advise on content and 

* SWAT meetings changes 


* Inform development of new content 
+ Work with DEEP to package 
* Poly and translations 
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Topics 


Top questions — As of 9am Feb 6th 


1. 


8. 
9. 
10. 


Empty — confusion at the top of the message — need to consider update - say voice, 
agent, representative 
354 Suspensions and fines to be top here. 
a. That journey would be trying to renew online and not able to due to fines and 
suspensions 
b. Question is what can be done at the application point to better manage that 
204 DriveTest 141 Apply? 
a. This is always a point of confusion for callers 
b. Drive Test line is busy — missed the date 
c. Need to ask CC about these calls 
221 Transfer a vehicle 108 Register Vehicle 
a. Complex and would like to know if they got the answer or went to an agent 
296 APP questions 
a. Does this call follow application issue? 
b. Is there something people are calling and asking? 
153 Renew Sticker 102 Renew Plate 
212 Drivers Licence Renew — expect this 
a. Priority fix needed as answers broken 
190 Digital Reminder 
93 Special Permit 
71 CVOR 


GovTechON 


topic 
DV_DL.SuspensionsFines 
DV_APP 

DV_PV.Transfer 
DV_DL.DriversLicence_Renew 
DV_DL.DriveTest 
DV_PRNE.DigitalReminder 
DV_PV.RenewSticker 
DV_DL.Apply 
DV_PV.RegisterVehicle 
DV_PV.RenewPlate 
DV_PV.SpecialPermit 
DV_DL.CVOR 
DV_DL.RenewalSession 
DV_PV.Personalized Plate 
DV_DL.ChangesAddress 


DV_PV.Permit_Replace 


Page 4 of 27 


(GSiC) 


Count 


Ontario Y 
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Proposed Action List (GSiC) 


Under review for prioritization 


e Punctuation — Remove question mark, exclamation mark (all 
punctuation) 
° Start with the top hits (suspension) 
° Spelling of licence - Q&A bot is now America need to move to “license” 
°- Add speak to talk 
¢ Representative, Agent, operator — all the same 
¢ Upfront message — make a change to improve the understanding 


* A — researching why? 


* No -they do not like the answer 
e Analyze where it is said 


ServiceOntario has not further input to this yet 
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Message Review & Recommendation (GSiC) 
Struggling with asking for an agent and customer service representative. 


Asking for help Current Message: Our telephone service is a virtual 
assistant that can answer simple questions and om 
Agent: x 
x P connect your to a support agent when needed. This pams 
- Asking representative, operator, human, : p 7 Tea | oor 
human being conversation may be monitored or recorded for quality | 

Talk - speak: Add assurance and training purposes. It is important you do 
- speak i not say sensitive information (such as your social ne 
: Need to, Can I, |need, With a, Toa insurance number or date of birth). Let’s get started! 
Upfront message: 


Recommended Change to Content asoman m- | 

Need feedback - cy) 

* You can ask your questions on this new phone í 
system, and connect to a representative when 
needed. This new system responds to your voice. 

* You do not need to share any personal information. 
In fact, we ask that you do not say your name, social 
insurance or date of birth to the phone system. 

* Calls are monitored or recorded for quality 
assurance - so we can make improvements. Let’s get 
started! 


- Creating more understanding 


What can we change in recording to create 

emphasis: 

- Pauses before agent is mentioned 

» Pause before next sentence 

What can we say: 

- Virtual assistant is only part of the system 
— not the known 

* Use system — and mention voice 

+ What is asked for most (Representative) 
agent? 
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Questions about reporting (GSiC) 


What uncertain about with data 


e Repeat and rephrase — can we make it - | did not get that can you 
rephrase that? 
* Invalid means the CIVR does not understand 
* Time outs — the caller has not said anything for 5 seconds 
* Max errors — 3 times not understood and send to an agent 
* Time outs do not include Max errors 
* Repeat — number of times asked to repeat 
* 7% going to contact centre — is there an error seems low 
+ Are we reading the data correctly — Chuck has the report 
e Not 7% 
e Numbers are in the threshold — need to take a look a calcuations 
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Issue with Question (GSiC) 


How do I renew my driver’s licence — results in Suspension of fines. 
Call in 416-849-7031 


Thursday — Day One Friday — Day Two 

How do I renew my driver’s Licence How do | renew my driver’s Licence 

1, Okay suspended DL 1. Okay renew DL 

2. Okay if you recently moved 2. Okay renew DL 

3. Okay if you recently moved I need to renew my DL 

I need to renew my DL 1. Okay expired DL 

1. Okay expired vehicle permit 2. Okay when should you renew licence plate 
2. OK expired DL 3. Okay expired DL 

3. OK expired DL I need help renewing my licence 

I need to help renewing my licence 1. Okay when should you renew licence plate 
1. Okay expired DL 2. Okay renewing L licence 


2. Renewing an L licence 
3. Renewing an L licence 
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MVP QuickSight Reports and Understanding (GSiC) 
How we start - will be a learning experience for how we go forward. 
What we expect to see What actions are we taking 
-+ CIVR calls — 500 to 1000 a day - Track items in Lists — Defect List 
* Calls directed to agents? ~33% redirects a day » Keywords may be added — threshold TBD 
(of the 15% to start) 


» Redirects — it sounds like you want information 
» Current IVR does not offer agent in a few on — call or look at 


flows — this is a change - Broadcast message updates or if needed 


» How callers many had multiple questions? - What can we do for future - backlog 


E Note:trends for. planning » Do we need to translate this differently 
+ What are they calling about - trend 
» Missed utterances may be higher initially 


+ Internal users — trying it out 
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Reports and Surveys (GSiC) 


Once we see the usage, behaviours, and caller stories - we may want to develop 
reports for different audiences. 


Ideas for Possible BI Reports Surveys for callers 

* Top 10 missed utterance daily to spot trends » Repeat the Pre-ClVR survey once it is fully 

» Peak time of day trends with averages etc. deployed 

- After hours trends -+ Have callers listened to the CIVR message and 


i ? 
» Chart of total calls and then transfers to CIVR ee eee 


and agents » Do they know more before they call now? 


» Answered, busy, failed or no answer trends or 
weekly or by QID 


» Time in queue to an agent, transfer time 


» Insights on Voice of the Customer 
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(GSiC) 


Accessing QuickSight Reports Quick Access 
= Find the APP 

Two Step Process DA 

1. Setup Account ud Quicksight 

2. Access to view ae ra 

reports/dashboards will be 

shared 
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CIVR Analytics & Reporting: Amazon QuickSight (GSiC) 
MVP dashboard offers reporting data that can be targeted to specific date ranges. 


é ¢ 


Amazon QuickSight delivers easy to Gd Quicksight 
understand insights through Q 
interactive dashboards allowing for 


Dashboards 


bnrooard T oasroosra 


l 


QnABotMetrics_Eng_Prod3. CIVR Navigational Report 


wi wi 


analysis of data. howeons Bowen 


Reporting data is essential for 
improving CIVR performance, which 
includes learning more about callers 
and resolving technical issues. 


S a 


RACE_REPORTDATAPROD | QnABotMetrics_Fr.P 


Dashboards 


QuickSight reports will assist K 
ServiceOntario and MTO with 8 
iterative improvements to the D&V 
CIVR needed for long term success. 


ntation Name C Ontario Y 


MPBSD 23-071 Record 14 DV-CIVR - Reporting Overview - Story - 2023-02-06 


Page 13 of 27 


Amazon QuickSight: Types of Reporting (GSiC 


1. QnA Bot Metrics 


2. Trace 


3. CIVR Navigation 


+ What: Provides data related to what the CIVR understand the callers are asking, what answers are being provided, what 
words the CIVR is not picking up, and the topics count for select period. 
+ Who: SO CCMI & CMI & MTO Policy/Program Areas 


+ Example: Leverage this report to compare what were the most frequently asked questions that are answered or missed 
utterances that are not. This can provide insight into what specific information customers are looking for from one month to 
another. ACTION: update webpages, and for continuous improvement. 


What: CVIR collects what it believes the callers' actions are while using the CIVR which provides understanding into caller 
behavior. Understanding how easy or difficult the are callers able to navigate the CIVR - if there are many timeouts or max 
invalids that could indicate the CIVR does not understand what the caller is asking, and they are having difficulty. 

+ Who: SO CCMI & CMI 


+ Example: Review the Trace report to see where callers are repeatedly asking CIVR to “repeat” the answers which could be 
an indication that the information presented is not clear or too much information is provided. ACTION: the specific 
script(s) should be reviewed and modified. 


e What: Navigational report which provides a summary of how callers moved through the CIVR - total times a questionID 
was referenced, hang-ups, timeouts, repeats, requests for agents, etc. for each Question ID. 
e Who: SO CCMI/CMI & MTO 


+ Example: ServiceOntario Locations QID was referenced 22 times which had 1 timeouts, 9 repeats and 2 hang-ups. This 
could indicate that the caller was satisfied with the information provided even though there were 9 repeats as the callers 
want to ensure they retain the correct information. Additional with no invalids, timeouts or max repeats/errors this could 
also indicates callers are receiving the information they required and had no further questions. ACTION: — use this report to 
see which QIDs are triggering the most requests for agent transfers and update those QIDs to provide more information. 
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(GSiC) 


View of Metrics, Monitoring & KPIs 


Digital Enterprise Engagement Practice: “Connecting the people and businesses of Ontario with their Government” 
Government Services Integration Cluster: “Enable the best experiences for citizens, business and employees” 
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1. QnA Bot Metrics (GSiC) 


View a dashboard for a time period selected. 
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1. System Monitoring — Questions Answered and Fulfilled 
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(GSiC) 


Learn about what people are asking most — Voice of the Customer. 


Reporting — Starting with similar reports, and word cloud for focus. 

+ Identify the most popular questions that the CIVR believes callers are asking 
within a certain time period 

+ This example indicates callers may have been inquiring about transferring 
ownership 

» Agent utterance was requested and fulfilled 42 within the time period 

+ “| want to pay a driver fine” was the second most popular fulfilled utterance 
indicating most callers are asking about this specific topic or QID 


Usage 
Start Date & End Date 
Shows total number of QIDs referenced 
Questions — Question Count 
Shows number of callers hitting the specific QID including 
main menu, agent and no matches 
Benefit: we can see what the CIVR thinks the callers are 
asking about 
Can sort the most important/popular QIDs 
Answer — Answer Cloud 
Answer scripts played for questions asked 
Provides a count of the most popular answers the CIVR is 
providing 
Fulfilled — Fulfilled Utterances Count 
Utterances that the CIVR was about to fulfill based upon 
the request it understood and provided the response 
Topics — Topics Count 
Can sort most important/popular topics the callers are 
calling about 
Each topic can have many specific QIDs associated with it 
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1. System Monitoring — Missed Utterances and Word Cloud (GSiC) 


Learn about the what people are asking — Voice of the Customer. 


Reporting 
» Missed utterances can be used to see what CIVR was not able to fulfill and provide an answer for 
* They can also be used to indicate what new questions can be added to the QnABots for the CIVR to recognize 


Missed Utterances Count 
Missed Utterances Count 


utterance Come 
utterance Count K 
i A : > “DV. ‘PV. Renewsticker 
car prices to mention Ov.DL-Apely 
abird 1 feedback 1 DVL DL. DriveTest DV_DL. Drivesticence Renew 
[OV_PV.Permit Replace DV_OL DiglalReminder w 
a one i need to pay 1 gibberish J emp Ly DLO 
a room 1 i have got my face thy 1 Ov. DL Suspensionsfines, DV_PVRegisterVehicle 


a time limit to submit my application to apply... 1 ihust myntrforthecard “T ihave got my ace]? DV_ PV. It n sfer r 


i know i hope it by this 


a well i need to apply for under you for twent... 1 DV “A P ONY. -speciatpermt 
jibrish 1 ter 

aape card 1 S "Dior Ronewatsecsion 
no 2 

sens 1 oh i need information on pocket permit for m... 

allergies russian 1 oh okay ummm 

am | eligible for an a p p 1 oh party but its for existing bilety needs Over time the data will be enriched 

Total 1,194 » Contact handling quantitative statistics will include 
Total 38 


speech analytics capabilities (for clients that use the EP- 
cloud instance) 
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2. Trace Report (GSiC) 


We can trace what people are doing and the time spent in the CIVR. 
Reporting Main Use 
» Which QIDs were presented to the caller 
-+ Potential time spent, invalids, calls going to agents, etc. 
» The number of requests for an agent - could indicate a change required to that QID 


(ad QuickSight 


Trace Reporting x 

* This is a view is available for a Gi a tnd ome 
requested time period 

* Each conversation is assigned 
a code 
Time stamp shows when the 
calls came in (UTC) 

-The conversation is outlined 
by the naming conventions 
for each item 

* Time is noted in number of 
seconds for each named 
convention 


id, and Alirequests 


AllRequests 


OLBEGINDLOS] 0.Globsl_Mandter MainMerdAgentia9s) 


wwcemebocd.serts0501) DV_DL DriveTest_Permanent 


ecumentsapent(24s} 0. Global SmallTalk AreYouThere14s} VOL OriveTest 


DL briveTes 


JV DL DoiveTest_Driversicencl 


katos] BV. OL DrlveTest_Canacle 


M194} Dv. 01.0 


OV DL.Agpty G08) OV DL. Apply. G8] OV. 


View: $00 items V 
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2. Trace Report more 
“Showing and telling” trace report value by example. 


Example: Hang-up 
-Caller reached CIVR (English) and then hung-up 


Contactid DateTime Civrid AllRequests 
86266c01-1bb0-4594-956f-65ca631981ca Nov 21, 2022 Timeout 0_BEGIN_E()[0s] 0_Global_Handler.MainMenu(invalid)[43s} 
bO8fc9e7-01b2-49d1-a2f3-40b1fc13cd46 Nov 21, 2022 2290 


Example: Successful Call 


+ Caller asked about suspension or fines and heard the answer script/information and hung-up 
Caller did not ask for an agent, repeat or any other questions 


€41131d1-cf23-4e83-9f21-f18726669aeb Nov 21, 2022 2289 0_BEGIN_E0[0s] DV_DL SuspensionsFines_info()[40s} 


Example: Request for Agent 
* Caller reached CIVR (English) and then hung-up 


16d07d0-8106-4b52-bd67-fabb764ffd7e Nov 21, 2022 2291 0_BEGIN_E()[0s] O_Global_Handler. MainMenu(Agent)[48s] 


Example: Did not get required information 
* Caller connects with CIVR (French), gets DriveTest_Appointment information three times and 
last attempt timeouts (did not say anything after the answer script was provided) 


5009bdb1-dacc-4075-ac27-5b5f504224c0 Nov22,2022 Timeout 
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(GSiC) 


Definitions 


GovTechON 


Contact ID: The unique identifier 
created from Amazon Connect 


Civrid: The custom ID created 
from Amazon Connect which is 
created every time a call goes 
from on-prem SO D&V IVR to 
CIVR in Amazon 


Timeout: If a call is made directly 
to CIVR it shows timeout, but if a 
call is made to On-prem solution 
first then it provides the CIVRID 
O_BEGIN: The start of all calls 
made into CIVR 

Global Handler.MainMenu: 
Caller utters invalid 


word/phrase, start over, | want 
to start over 


0.BEGIN_FO[0:] DV_OLOriveTest Appointment FO[SE«] DV_DL DriveTest_ Appointment _Flinvalid)(22s] DV_DL OriveTest_ Appointment F(Timecut)[16:) 


Ontario Y 


MPBSD 23-071 Record 14 DV-CIVR - Reporting Overview - Story - 2023-02-06 Page 20 of 27 


3. CIVR Navigational Report (GSiC) 


We can review a summary of how callers navigated the CIVR. 


Reporting Main Use 
Provides total number of callers and average time spent in the CIVR (displayed in seconds) 
Summary of calls made, total times a questionID (QID) was referenced and specific information on each QID (hang-ups, 
timeouts, repeats, requests for agents, etc.) 
If we want to reduce calls being transferred to agent — use this report to see which QIDs are triggering the most requests for 
agent transfers 


This is a view is pulled for a 
requested time period 
Total number of calls - 12 calls 332 167 165 59.23 
made it to the “Apply_Cost_Info 
QiD” 
The breakdown of 12 calls: 

1 timeout 

1 invalid 

5 repeats 

1 agent request 
Multiple occurrences within a 
single call causes the total and 
breakdown to not equal 


Total Number of callers Number of English cater Number of French callers Average time spent in CIVR (See) 


Note: Columns are similar to the current IVR navigation reports 
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3. CIVR Navigational Report — Hang-up Calls 
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(GSiC) 


Each a topic/queue is identified under question ID — Data helps to understand the story. 


Queue ID Reporting example 

+ Hang-up calls — should we be 
concerned? 
2 calls and 2 hang-ups for 
QID APP address change 
No request to speak to an 


agent, repeat or error out 
The story we can tell here is 
that likely this indicates the 
callers are receiving the 
information they were 
seeking and hung-up 

No concern or action needed 


GovTechON 
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3. CIVR Navigational Report — Max Error (GSiC) 


Question ID data — needs to be understood in the context volumes for action. 


Queue ID Reporting example 

» Max Error — should we be concerned? 
QID was requested 17 times - the CIVR 
believes this for the QID for APP 
Address Change Sanaa 
1 MaxError — is noted which means a QuestiontD pate Toul | Timeouts | braids | epost | siaii | Apem® | sanapas | Daars | [Hecgnp 
caller uttered something that the CIVR ; p ; ; F 3 
could not understand 3 times for QID 
The story we can tell here is that 1 in 17 
callers was not understood by the CIVR 
and they were taken to the agent 


queue. ) OV.PyRenewsticer insurancevertted + 
This is not the primary desired 
experience, however being directed to 
the agent ensures and an answer 

No concern 

Missed utterances can be checked — 
though some volume may be required 
for trends 


J DV_PV.ReplaceLoststolendamagePt 2 2 7 
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KPI Measures (GSiC) 


Success is about more than delivering a project on time and on budget — it is about 
meeting our callers need for service, both to self-serve and to reach an agent. 


[Measure [Descente Saret | 


Baseline survey 
* caller is asked if they are willing to participate in a survey 


Customer + if they agree they are able to confirm their phone number 72.40% 80% 
satisfaction * The system places an outbound call to the caller to conduct the survey j 
(overall) CIVR — Relaunch Survey 


* Callback to callers after the call (80% Standard) - approvals required 
* CIVR survey tool to connect to be developed 
Speed to need Baseline - time for callers to reach first topic in IVR 


(Time to first CIVR - time for callers to reach first topic 112 sec 100.8 sec 
topic) * compare against baseline to measure improvement. (-10%) 

Baseline - time for callers to reach MMR in IVR by pressing 0 
Speed to need ciyp - time for callers to reach MMR 42 sec S7 aea 


(Timeto MMR) . compare against baseline to measure improvement (-10%) 


Baseline - measured in IVR (hang-up /total callers) 
% Self serve Any "hang-up" in a topic will be considered a self-serve 37% 33.3 % 
(% hang ups)  CIVR. Aspirational it would increase, but success is maintaining (-10%) 
Request Agent Baseline - number of callers that press 0 from top menu/total calls 7 A 
(%top menu) CIVR - number of callers that ask to speak to an agent immediately (-10%) 10% 9% 


Ontario Y 


MPBSD 23-071 Record 14 DV-CIVR - Reporting Overview - Story - 2023-02-06 Page 24 of 27 


(GSiC) 


QUESTIONS 
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Amazon QuickSight & EP iceReports (GSiC) 


The caller experience and insights from reports. 


AWS CIVR Reporting Enterprise Platform On-Prem iceReports 
* Callers experience with the CIVR Solution * Callers experience after being transferred to an 
before agent transfer agent - survey 
* Insights for business: * Insights on users/agents experience: 
* Caller trends, popular requests and caller * Number of calls handled by agents, calls 
preferences transferred, etc. 
* Insights for operations: * Insights on queue activity: 
* Identify potential problems that could * Number of calls that were in queue (avg. wait 
impact customer experience times, abandon time, on call, etc.) — need 


, . volume for impact. 
e Review performance metrics 
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Amazon QuickSight & ECCS iceReports Flow 


The caller experience and flow for reports. 


Enterprise Platform AWS 
Conversational IVR 


AWS QuickSight Reporting 


Main menu 


EN 
(Cove star} —H “Front end message you caling about 
oren i 


driver s license?” 


Callers question(s) 


Lets get started. Ask a question or say what 


For example, you can say "How do I renew my | 


Enterprise Platform 
On-Prem ice Soultion 


Sends calito ECCS 


>| answered, but 
wants confirmation 


26 Presentation Name 


| ECCS iceReports 


[Connects 
| Calle placed into | | x 
ith agent using sal Clients questions) 
49} aueue for an net H $ > lends 
svalableagent, | | ECCS omprem ICE answered tai 
platform 
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Defect List (GSiC) 
This needs to be used 100% of the time to fully understand and plan. 


Transparent Tracking 

* Daily planner was used for chatbot — we have decided to try a new route 
+ This will include actual defects and who it is assigned to 

* The features can grow based on need - agile 


iii Microsoft Lists 


kie Edit in grid view 1È Share v $ Export tograte All tems 
as Conversational IVR 
Defect List * © 
g 10 Utterance said (what the tester said ver. Expected QID R. Actual Response Priority Status Comments Add column 


Ontario Y 


